[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

vvchernov · 2022-01-12T10:37:16Z

Rename all mentions of STVM to TVM in the onnx runtime codebase.

Corrections were done after PR#10241 and PR#10211 were merged.

Notes:

Nuphar and TVM EPs building were separated in build and cmake files for convenient further work.
New key is used for build TVM EP +Cuda to avoid confusion with CUDA EP. Adding the new key is associated with the following --use_cuda usage issues:
Using the --use_cuda flag means that when building a project, you need to build CUDA EP. This is not the right approach if we only want to work with TVM EP. Building CUDA EP is a separate flow that requires its own nuances and dependencies, which are redundant if we only need TVM EP.
At the build.py script level, there are checks for --cuda_home and --cudnn_home flags, which are not needed for TVM. Also, setting the --use_cuda flag turns on a lot of additional logic when running build.py that is not needed for the TVM EP. In order to disable this additional logic, it is necessary to extend conditions if args.use_cuda: to if args.use_cuda and not args.use_tvm in many places. This is not conceptually correct, since these changes for TVM EP need to be made in methods that are specific to CUDA EP.
Additionally, there are problems using TVM EP via PYTHONPATH and the wheel package. This is because setup.py only supports one EP at a time. If we set --use_cuda and --use_tvm flags, then only one wheel package for CUDA EP will be built, because this is how the logic of working with providers in setup.py is arranged. Also, the condition for the _ld_preload.py extension for TVM EP will not be fulfilled, with the help of which the
If input shapes are unset and dynamic the ONNX runtime error is raised instead of warning and automatical inferred of dynamic dimension to 1.
There are two independent structures with the same name (TVMFuncState) in TVM EP internal code and in tests for TVM prepared by NUPHAR.

Hello @xadupre, can you recheck our updates after your PR?

tmoreau89 · 2022-01-14T00:53:33Z

@xadupre let us know if you have feedback on the changes we applied to the Nuphar TVM strings that conflicted.

xadupre · 2022-01-14T11:10:55Z

docs/python/inference/notebooks/onnxruntime-tvm-tutorial.ipynb

      "convert layout to NHWC: 0\n",
      "input tensor names: \n",
      "input tensor shapes: \n",
-      "Build TVM graph executor\n",
-      "****************** Success! ******************\n"
+      "/home/agladyshev/workspace/tvm/python/tvm/relay/build_module.py:414: UserWarning: target_host parameter is going to be deprecated. Please pass in tvm.target.Target(target, host=target_host) instead.\n",


Same warning as the removed one but is there any way to remove it?

That should be cleaned up; CC @KJlaccHoeUM9l and @vvchernov

xadupre · 2022-01-14T11:16:38Z

cmake/CMakeLists.txt

@@ -59,7 +59,7 @@ option(onnxruntime_BUILD_OBJC "Build Objective-C library" OFF)
 option(onnxruntime_USE_PREINSTALLED_EIGEN "Use pre-installed EIGEN. Need to provide eigen_SOURCE_PATH if turn this on." OFF)
 option(onnxruntime_BUILD_BENCHMARKS "Build ONNXRuntime micro-benchmarks" OFF)

-option(onnxruntime_USE_TVM "Build TVM for code generation." OFF)
+option(onnxruntime_NUPHAR_USE_TVM "Build TVM for code generation." OFF)


Why onnxruntime_NUPHAR_USE_TVM and not onnxruntime_USE_NUPHAR_TVM?

@xadupre it does not metter for me, it separated 'USE_TVM' after 'USE_STVM' renaming and 'USE_TVM' used by Nuphar. if you wish I can redo it

Every other similar constant starts with onnxruntime_USE.... It seems better to follow the same pattern.

tmoreau89 · 2022-01-19T01:43:45Z

@xadupre it appears the CI are blocked, would you mind unblocking? thank you.

edgchen1 · 2022-01-19T17:56:24Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

edgchen1 · 2022-01-19T17:56:47Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-01-19T17:57:16Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2022-01-19T17:57:27Z

Azure Pipelines successfully started running 8 pipeline(s).

vvchernov · 2022-01-20T14:23:04Z

Hello @xadupre and @edgchen1, I've fixed issues related to CI tests failure, can you review and restart the tests

tmoreau89 · 2022-01-20T18:30:18Z

Recommendation is to land this PR after #10211 is merged

tmoreau89 · 2022-02-04T16:40:32Z

@xadupre and @edgchen1 - we have finished rebasing and fixing some of the build issues we uncovered after #10211 merged. Please see the updated PR description.

Would it be possible to re-trigger the CI and get an additional set of reviews? Thanks

xadupre · 2022-02-04T16:42:35Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2022-02-04T16:43:30Z

Azure Pipelines successfully started running 9 pipeline(s).

cmake/CMakeLists.txt

xadupre · 2022-02-04T16:47:41Z

docs/TVM_EP.md

 apt-get install -y python3 python3-dev python3-pip python3-setuptools gcc libtinfo-dev zlib1g-dev build-essential cmake libedit-dev libxml2-dev llvm-12
 pip3 install numpy decorator attrs
 ```

-Clone this repo.
+If you want to use the `GPU` then make sure you have installed NVidia driver and CUDA Toolkit.


If TVM has specific requirement on the version, is there a page the documentation can link to?

Hello @xadupre!
There are no special version requirements.
There are also no additional links to documentation. CUDA is an external dependency in this context, so it is up to the user to decide how they will deliver this dependency.

Internally we're using CUDA 10, but moving over to CUDA 11.1 and ensuring this leads to no performance regressions.

xadupre · 2022-02-04T16:49:09Z

docs/TVM_EP.md

-pip3 uninstall onnxruntime onnxruntime-stvm tvm -y
-whl_path=$(find ./build/Linux/Release/dist -name "*.whl")
+python3 -m pip uninstall tvm -y
+whl_path=$(find ./build/<OS_NAME>/Release/_deps/tvm-src/python/dist -name "*.whl")


Good catch. linux or os_name is not mandatory. It can be removed.

We assume that ONNX runtime is built by build.sh (see TVM_EP.md). Inside the latter the path with OS_DIR is constructed due to this linux or os_name is mandatory

docs/TVM_EP.md

docs/python/inference/notebooks/onnxruntime-tvm-tutorial.ipynb

tools/ci_build/build.py

tmoreau89 · 2022-02-07T17:19:20Z

@xadupre please let us know if you have remaining outstanding questions

vvchernov · 2022-02-09T13:55:51Z

Hello @xadupre , could you recheck our changes after the last review and restart CI tests?

…r in cmake and build files

vvchernov · 2022-02-11T17:38:49Z

Good evening @xadupre. PR #10524 was merged, I rebased our branch onto master, please start CI tests

xadupre · 2022-02-11T18:08:12Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

xadupre · 2022-02-11T18:08:36Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-02-11T18:09:02Z

Azure Pipelines successfully started running 9 pipeline(s).

azure-pipelines · 2022-02-11T18:09:22Z

Azure Pipelines successfully started running 8 pipeline(s).

vvchernov · 2022-02-12T08:48:54Z

PR #10534 and #10533 have the same CI failure. Waiting for fix by somebody

tmoreau89 · 2022-02-14T19:09:22Z

@xadupre - is the issue found in orttraining-amd-gpu-ci-pipeline a known issue? Currently blocked on getting the CI to go green; seems unrelated to this PR.

edgchen1 · 2022-02-14T22:29:00Z

/azp run orttraining-amd-gpu-ci-pipeline

azure-pipelines · 2022-02-14T22:29:11Z

Azure Pipelines successfully started running 1 pipeline(s).

vvchernov · 2022-02-15T07:53:38Z

Hello @xadupre! Please check last time and approve the PR

…icrosoft#10260) * update java API for STVM EP. Issue is from PR#10019 * use_stvm -> use_tvm * rename stvm worktree * STVMAllocator -> TVMAllocator * StvmExecutionProviderInfo -> TvmExecutionProviderInfo * stvm -> tvm for cpu_targets. resolve onnxruntime::tvm and origin tvm namespaces conflict * STVMRunner -> TVMRunner * StvmExecutionProvider -> TvmExecutionProvider * tvm::env_vars * StvmProviderFactory -> TvmProviderFactory * rename factory funcs * StvmCPUDataTransfer -> TvmCPUDataTransfer * small clean * STVMFuncState -> TVMFuncState * USE_TVM -> NUPHAR_USE_TVM * USE_STVM -> USE_TVM * python API: providers.stvm -> providers.tvm. clean TVM_EP.md * clean build scripts #1 * clean build scripts, java frontend and others #2 * once more clean #3 * fix build of nuphar tvm test * final transfer stvm namespace to onnxruntime::tvm * rename stvm->tvm * NUPHAR_USE_TVM -> USE_NUPHAR_TVM * small fixes for correct CI tests * clean after rebase. Last renaming stvm to tvm, separate TVM and Nuphar in cmake and build files * update CUDA support for TVM EP * roll back CudaNN home check * ERROR for not positive input shape dimension instead of WARNING * update documentation for CUDA * small corrections after review * update GPU description * update GPU description * misprints were fixed * cleaned up error msgs Co-authored-by: Valery Chernov <[email protected]> Co-authored-by: KJlaccHoeUM9l <[email protected]> Co-authored-by: Thierry Moreau <[email protected]> (cherry picked from commit 1cdc23a)

vvchernov changed the title ~~[TVM EP] Rename Standalone TVM (STV) Execution Provider to TVM EP~~ WIP: [TVM EP] Rename Standalone TVM (STV) Execution Provider to TVM EP Jan 12, 2022

vvchernov changed the title ~~WIP: [TVM EP] Rename Standalone TVM (STV) Execution Provider to TVM EP~~ WIP: [TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP Jan 12, 2022

xadupre mentioned this pull request Jan 13, 2022

STVM, NUPHAR, remove tvm from submodules list, checks pointers are not null. #10211

Merged

vvchernov mentioned this pull request Jan 14, 2022

[ONNX RUNTIME] [TVM EP]: rename STVM to TVM microsoft/onnxruntime-extensions#193

Closed

xadupre reviewed Jan 14, 2022

View reviewed changes

xadupre previously approved these changes Jan 14, 2022

View reviewed changes

vvchernov changed the title ~~WIP: [TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP~~ [TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP Jan 16, 2022

vvchernov mentioned this pull request Jan 20, 2022

[TVM EP] support of TVM Virtual Machine #10341

Merged

vvchernov dismissed xadupre’s stale review via 6e16164 January 20, 2022 14:20

vvchernov force-pushed the vc/rename branch from 6e16164 to 574fb92 Compare February 2, 2022 11:21

xadupre reviewed Feb 4, 2022

View reviewed changes

cmake/CMakeLists.txt Outdated Show resolved Hide resolved

xadupre reviewed Feb 4, 2022

View reviewed changes

docs/TVM_EP.md Outdated Show resolved Hide resolved

xadupre reviewed Feb 4, 2022

View reviewed changes

docs/python/inference/notebooks/onnxruntime-tvm-tutorial.ipynb Show resolved Hide resolved

xadupre reviewed Feb 4, 2022

View reviewed changes

tools/ci_build/build.py Outdated Show resolved Hide resolved

Valery Chernov and others added 11 commits February 11, 2022 20:36

small fixes for correct CI tests

61673d6

clean after rebase. Last renaming stvm to tvm, separate TVM and Nupha…

dd255dd

…r in cmake and build files

update CUDA support for TVM EP

5da0db9

roll back CudaNN home check

324c037

ERROR for not positive input shape dimension instead of WARNING

39f18e5

update documentation for CUDA

50b4a47

small corrections after review

f798af1

update GPU description

94b876b

update GPU description

0ed6ed5

misprints were fixed

48e5400

cleaned up error msgs

1a2592e

vvchernov force-pushed the vc/rename branch from 36881c0 to 1a2592e Compare February 11, 2022 17:36

KJlaccHoeUM9l approved these changes Feb 15, 2022

View reviewed changes

xadupre approved these changes Feb 15, 2022

View reviewed changes

xadupre merged commit 1cdc23a into microsoft:master Feb 15, 2022

vvchernov mentioned this pull request Feb 15, 2022

[DOC] update TVM EP docs #10562

Merged

vvchernov deleted the vc/rename branch February 25, 2022 08:49

skottmckay mentioned this pull request Apr 11, 2022

bug need to solve #11166

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

vvchernov commented Jan 12, 2022 •

edited

Loading

tmoreau89 commented Jan 14, 2022

xadupre Jan 14, 2022

tmoreau89 Feb 10, 2022

xadupre Jan 14, 2022

vvchernov Jan 14, 2022 •

edited

Loading

xadupre Jan 14, 2022

vvchernov Jan 14, 2022

tmoreau89 commented Jan 19, 2022

edgchen1 commented Jan 19, 2022

edgchen1 commented Jan 19, 2022

azure-pipelines bot commented Jan 19, 2022

azure-pipelines bot commented Jan 19, 2022

vvchernov commented Jan 20, 2022

tmoreau89 commented Jan 20, 2022

tmoreau89 commented Feb 4, 2022

xadupre commented Feb 4, 2022

azure-pipelines bot commented Feb 4, 2022

xadupre Feb 4, 2022

KJlaccHoeUM9l Feb 7, 2022

tmoreau89 Feb 7, 2022

xadupre Feb 4, 2022

vvchernov Feb 7, 2022

tmoreau89 commented Feb 7, 2022

vvchernov commented Feb 9, 2022

vvchernov commented Feb 11, 2022

xadupre commented Feb 11, 2022

xadupre commented Feb 11, 2022

azure-pipelines bot commented Feb 11, 2022

azure-pipelines bot commented Feb 11, 2022

vvchernov commented Feb 12, 2022

tmoreau89 commented Feb 14, 2022

edgchen1 commented Feb 14, 2022

azure-pipelines bot commented Feb 14, 2022

vvchernov commented Feb 15, 2022

[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

[TVM EP] Rename Standalone TVM (STVM) Execution Provider to TVM EP #10260

Conversation

vvchernov commented Jan 12, 2022 • edited Loading

tmoreau89 commented Jan 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvchernov Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmoreau89 commented Jan 19, 2022

edgchen1 commented Jan 19, 2022

edgchen1 commented Jan 19, 2022

azure-pipelines bot commented Jan 19, 2022

azure-pipelines bot commented Jan 19, 2022

vvchernov commented Jan 20, 2022

tmoreau89 commented Jan 20, 2022

tmoreau89 commented Feb 4, 2022

xadupre commented Feb 4, 2022

azure-pipelines bot commented Feb 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tmoreau89 commented Feb 7, 2022

vvchernov commented Feb 9, 2022

vvchernov commented Feb 11, 2022

xadupre commented Feb 11, 2022

xadupre commented Feb 11, 2022

azure-pipelines bot commented Feb 11, 2022

azure-pipelines bot commented Feb 11, 2022

vvchernov commented Feb 12, 2022

tmoreau89 commented Feb 14, 2022

edgchen1 commented Feb 14, 2022

azure-pipelines bot commented Feb 14, 2022

vvchernov commented Feb 15, 2022

vvchernov commented Jan 12, 2022 •

edited

Loading

vvchernov Jan 14, 2022 •

edited

Loading